Hierarchical RL with Learned World Graph for Navigation
Project information
- Category: Artificial Intelligence
- Focus: Reinforcement Learning Unsupervised Learning Variational Autoencoders Hierarchical RL
- Tech Stack: Pytorch Gymnasium (Minigrid) NumPy Matplotlib
- Project date: Nov 2025 , Rebuilt in 2026
- Official Repository
72.5% reached all 5 goals
86.8% reached 4+ goals
Success Rate in 10,000 Steps
(5,000 Ep., Across 5 layouts)
4.87
Avg Reward out of 5.59 Theoretical Max
(5,000 Ep., Across 5 layouts)
3,656 Steps / 61.1 s
Avg Steps / Seconds to Completion
(5,000 Ep., Across 5 layouts)
(10k steps counted if not completed)
Overview
Reimagined and rebuilt from scratch, inspired by Shang et al. (2019). The original paper proposes a general HRL framework; this project reconstructs and substantially extends its core ideas for the MultiGoal task, introducing an original Wide-then-Narrow goal-selection scheme, a three-state Worker FSM, and a full pretraining pipeline developed from scratch.
A Hard-Kumaraswamy VAE explores the environment unsupervised, identifying topological bottleneck states — pivotal states — used to build a World Graph of reachable transitions.
A structured pretrain pipeline then trains a graph-aware Worker and a two-headed Manager independently, before integrating them: the Manager selects high-level goals from the graph; the Worker reaches them through a finite-state machine combining Dijkstra graph traversal and local MLP navigation.
Key Elements
Pivotal State Discovery
An unsupervised process to detect informative states that capture the structural essence of the environment.
Structured Pretrain Pipeline
Worker trained on progressive radius stages with Prioritized Experience Replay; edges refined post-pretrain. Manager Wide and Narrow heads pretrained independently before integration.
World Graph Construction
$$\mathcal{G} = (\mathcal{P}, \mathcal{E})$$
Building a task-agnostic map with pivotal states and feasible transitions
Wide-then-Narrow Goal Selection
$\mathcal{L} = \mathbb{E}[\hat{A}\log\pi_\theta(a|s,g)] - \beta\,\mathcal{H}[\pi_\theta]$
A2C-LSTM Manager picks a graph pivot (wide goal), then a Manhattan diamond cell around it (narrow goal). Worker reaches it via FSM: Dijkstra traversal, then local MLP navigation.
Contacts
Get in touch with me!